Summary

SharedConcurrencyLimitTest has been extremely flaky recently. The issue appears to be from the fact that:

1. We're running on a resource constrained system.
2. This test runs multiple concurrent traces (3), while also triggering cancellations. This is somewhat intensive inside the test environment.
3. Because of the above 2 reasons, the test app fails its responsiveness check (in Connect mode, taking longer than 3 seconds to respond) when doing additional API calls, causing a test failure.

To fix it, add retry to our API calls in these tests when it appears that the process has failed its liveliness check. For now I've only introduced this retry logic to EgressTests. If we find other tests are facing this same issue we can consider moving this retry logic further down inside of our api client.

Alternatively, the new pruning algorithm in listen mode should prevent this issue. Once it's out of its experimental phase, we can consider moving these tests over to Listen mode to leverage the new algorithm and removing the retries.

Release Notes Entry